Skip to content

RAII things - #759

Draft
cds-amal wants to merge 5 commits into
solana-foundation:mainfrom
cds-rs:spike/storage-backend
Draft

RAII things#759
cds-amal wants to merge 5 commits into
solana-foundation:mainfrom
cds-rs:spike/storage-backend

Conversation

@cds-amal

Copy link
Copy Markdown
Contributor
counter baseline HEAD
on-disk: pools created 10, plus 180 cache reuses 10
on-disk: connections opened / closed 100 / 0 100 / 100
on-disk: connections live after the last drop 100 0
in-memory: pools created 190 10
in-memory: connections opened / closed 1,900 / 1,900 100 / 100
peak live connections 290 10

Process-wide atomic counters for pool and connection events, a
snapshot/since API, and an ignored census_workload test that builds and
drops ten surfnets per phase and prints the movement. Nothing is
connected yet; the counters read zero until a backend routes its pool
construction and connections through them. Closes are counted by Drop on
a connection wrapper around the r2d2 manager, because
CustomizeConnection::on_release fires only for broken or reaped
connections, not at pool drop, and undercounts.
SQLite pools now build through CountingSqliteManager, and the build sites
in get_or_create_shared_pool report creation and cache reuse.

census_workload, ten surfnets built and dropped per phase:

    on-disk:   pools 10 (+180 reuses), conns opened 100, closed 0,
               100 still live after the last drop
    in-memory: pools 190, conns opened 1,900, closed 1,900,
               peak 190 live during one surfnet

Connections owned by the process-global SHARED_POOLS survive every
surfnet drop. The in-memory line prices the 19 isolated pools each
surfnet builds.
…ions

Each surfnet now opens one StorageBackend from its database URL and mints
all of its kv stores from it. The SQLite pool moves off the process-global
SHARED_POOLS map onto the backend, so connections live exactly as long as
the surfnet that opened them; the checkpoint-dedup set and the per-store
shutdown fan-out fall away with it. PostgreSQL keeps its process-level
pool cache behind the backend, where pooling actually amortizes a network
connection. In-memory SQLite now shares one database per surfnet instead
of building 19 isolated pools, matching the on-disk layout of distinct
tables in one database.

The counting manager remains the pool's connection type, so connection
opens and closes stay counted; the census build-site hook is reconnected
in the next commit.
SqliteBackend::open reports pool creation, and a new assertion test,
dropping_backend_closes_connections, fails if any connection opened by a
backend survives its drop.

census_workload, same workload as the baseline measurement two commits
back:

    on-disk:   pools 10, conns opened 100, closed 100, live 0
    in-memory: pools 10, conns opened 100, closed 100, live 0

Ten dropped surfnets leave 100 connections live on the baseline and none
here; in-memory construction opens 190 connections per surfnet there and
10 here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant